Add methods for getting bytes + json to store abc by d-v-b · Pull Request #3638 · zarr-developers/zarr-python

d-v-b · 2026-01-04T16:34:11Z

Adds some convenience methods to the store abc:

get_bytes_async
get_json_async
get_bytes
get_json

The async methods have a _async suffix. get_bytes and get_json are synchronous.

this PR allows the following, which is a huge QOL improvement for people working with Zarr stores.

import zarr
from zarr.storage import MemoryStore

store = MemoryStore({})
zg = zarr.create_group(store)

zmeta = store.get_json('zarr.json')
print(zmeta)
# {'attributes': {}, 'zarr_format': 3, 'node_type': 'group'}

codecov · 2026-01-04T16:44:29Z

Codecov Report

❌ Patch coverage is 69.33333% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.99%. Comparing base (acdd892) to head (bf5e635).

Files with missing lines	Patch %	Lines
src/zarr/abc/store.py	53.33%	7 Missing ⚠️
src/zarr/testing/store.py	76.92%	6 Missing ⚠️
src/zarr/storage/_local.py	70.58%	5 Missing ⚠️
src/zarr/storage/_memory.py	70.58%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3638      +/-   ##
==========================================
+ Coverage   60.90%   60.99%   +0.09%     
==========================================
  Files          86       86              
  Lines       10174    10244      +70     
==========================================
+ Hits         6196     6248      +52     
- Misses       3978     3996      +18

Files with missing lines	Coverage Δ
src/zarr/storage/_local.py	`70.96% <70.58%> (+0.37%)`	⬆️
src/zarr/storage/_memory.py	`58.97% <70.58%> (+2.53%)`	⬆️
src/zarr/testing/store.py	`75.37% <76.92%> (+0.37%)`	⬆️
src/zarr/abc/store.py	`38.31% <53.33%> (+1.43%)`	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…into feat/get_json

paraseba · 2026-01-04T18:34:27Z

What is the advantage of making these helpers part of the ABC, instead of helper functions or a new concrete Store? Is the intention that certain implementations will override these? If so, what would be the advantages of overriding?

d-v-b · 2026-01-04T20:48:05Z

What is the advantage of making these helpers part of the ABC, instead of helper functions or a new concrete Store?

Putting this functionality on the ABC ensures that all stores implement it, without making per-store changes. Reading JSON is not actually new functionality, and ultimately I want to use these helpers internally (specifically, the async ones for reading JSON). I didn't do that in this PR because it requires changes to the StorePath class.

Is the intention that certain implementations will override these? If so, what would be the advantages of overriding?

Yes, in this PR the localstore and memorystore implementations override these to allow prototype to be None. See https://github.com/d-v-b/zarr-python/blob/d70a5e5277ca7a225d9fb0ac22945b5e1fcb8cc3/src/zarr/storage/_memory.py#L178-L184, and I'm following the pre-existing convention on those stores with these overrides.

The advantage in this case is to work around the cumbersome requirement that store methods take a prototype parameter, which IMO we should move away from, but that's something for another PR

d-v-b · 2026-01-05T09:28:55Z

src/zarr/abc/store.py

        ...

+    async def get_bytes_async(
+        self, key: str, *, prototype: BufferPrototype, byte_range: ByteRequest | None = None


since the output of this method is a bytes object (CPU memory), I don't see the value of specifying a prototype in this method. can anyone suggest a situation where specifying the prototype would be useful here?

cc @TomAugspurger

I think I agree with you, that a prototype probably isn't needed. But I'd suggest including it for consistency with the other methods.

TomAugspurger

Since methods like get are asynchronous, I'd recommend making these convenience methods async-first and add a _sync suffix for the sync versions. get_bytes / get_bytes_async, etc.

TomAugspurger · 2026-01-05T13:44:29Z

src/zarr/abc/store.py

        ...

+    async def get_bytes_async(
+        self, key: str, *, prototype: BufferPrototype, byte_range: ByteRequest | None = None


I think I agree with you, that a prototype probably isn't needed. But I'd suggest including it for consistency with the other methods.

TomAugspurger · 2026-01-05T13:46:56Z

src/zarr/abc/store.py

+        Examples
+        --------
+        >>> store = MemoryStore()
+        >>> store.set("data", Buffer.from_bytes(b"hello world"))


I think this needs an await.

TomAugspurger · 2026-01-05T13:47:30Z

src/zarr/abc/store.py

+        ----------
+        key : str, optional
+            The key identifying the data to retrieve. Defaults to an empty string.
+        prototype : BufferPrototype


We'll want to align this with what we do in the async version.

TomAugspurger · 2026-01-05T13:49:55Z

src/zarr/storage/_local.py

+        >>> print(data)
+        b'hello'
+        """
+        if prototype is None:


Can't this be done in the base class, and then all the child classes reuse the same implementation?

I considered this direction. It would definitely be more convenient, but coupling the abstract store definition to the behavior of the default_buffter_prototype(), which in turn depends on the global config, seems much worse than other options (namely, baking buffer type into the store definition, or into store creation)

d-v-b · 2026-01-05T19:12:33Z

Since methods like get are asynchronous, I'd recommend making these convenience methods async-first and add a _sync suffix for the sync versions. get_bytes / get_bytes_async, etc.

@jhamman had the same preference and I haven't heard anyone strongly support the current state of this PR, so I'll make this change

d-v-b · 2026-01-08T11:31:02Z

get_json and get_bytes are now async, and get_json_sync and get_bytes_sync are now synchronous.

d-v-b · 2026-01-16T11:11:55Z

in the dev meeting this week @maxrjones suggested making the new methods private API, so:

store._get_json_sync
store._get_json
store._get_bytes
store._get_bytes_sync

Is that an accurate summary max?

codspeed-hq · 2026-01-16T11:20:25Z

CodSpeed Performance Report

Merging this PR will not alter performance

_{Comparing d-v-b:feat/get_json (89c5acc) with main (b30ae18)}

Summary

✅ 48 untouched benchmarks
⏩ 6 skipped benchmarks¹

6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

maxrjones · 2026-01-16T17:41:38Z

in the dev meeting this week @maxrjones suggested making the new methods private API, so:
store._get_json_sync
store._get_json
store._get_bytes
store._get_bytes_sync
Is that an accurate summary max?

this was my off-hand reaction, but I am glad to defer to those with better API design instincts

d-v-b · 2026-01-16T18:08:24Z

If it makes this PR more palatable, I'm happy making stuff private for now!

d-v-b · 2026-01-20T09:36:35Z

the new methods are now all private. Since "read JSON" is indisputably a core behavior of a Zarr store, I think we should make these public sooner rather than later.

d-v-b · 2026-01-23T14:29:36Z

As the methods are private, I removed the release notes, so I think this is ready for final review

maxrjones

The prototype argument is awkward IMO, but seems worthwhile to merge this and try it out more to see whether that awkwardness matters.

maxrjones

The prototype argument is awkward IMO, but seems worthwhile to merge this and try it out more to see whether that awkwardness matters.

d-v-b · 2026-01-26T08:43:29Z

The prototype argument is awkward IMO

Agreed but that's inherited awkwardness -- we have to call store.get with these methods, and store.get (awkwardly) takes the prototype as a required positional argument. It gets a bit less awkward with #3644

add store routines for getting bytes and json

021bd44

github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 4, 2026

d-v-b added 5 commits January 4, 2026 17:46

Merge branch 'main' into feat/get_json

61cf6d0

check for FileNotFoundError when a key is missing

7d26b8e

remove storepath methods

971c3e4

Merge branch 'feat/get_json' of https://github.com/d-v-b/zarr-python …

b7f7e38

…into feat/get_json

changelog

d70a5e5

github-actions bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Jan 4, 2026

d-v-b commented Jan 5, 2026

View reviewed changes

TomAugspurger reviewed Jan 5, 2026

View reviewed changes

d-v-b added 2 commits January 8, 2026 10:37

rename methods

a213058

continue renaming / test refactoring

38ff517

d-v-b mentioned this pull request Jan 8, 2026

Default buffer logic in store ABC #3643

Open

d-v-b added 2 commits January 8, 2026 11:47

refactor new test functions

bdc4ef8

Merge branch 'main' into feat/get_json

e8ca484

d-v-b added 2 commits January 8, 2026 13:46

Merge branch 'main' into feat/get_json

0a97eb4

Merge branch 'main' into feat/get_json

5d2018d

d-v-b mentioned this pull request Jan 8, 2026

default buffer logic #3644

Open

Merge branch 'main' into feat/get_json

bf5e635

d-v-b requested a review from TomAugspurger January 9, 2026 09:16

Merge branch 'main' into feat/get_json

6f71a6b

d-v-b added 3 commits January 20, 2026 10:31

Merge branch 'main' into feat/get_json

d09c072

make new methods private

d259a7e

remove changelog entry for private API

b0e7ebe

github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 20, 2026

Merge branch 'main' into feat/get_json

e7a98f2

Merge branch 'main' into feat/get_json

89c5acc

maxrjones approved these changes Jan 26, 2026

View reviewed changes

d-v-b merged commit 0e61449 into zarr-developers:main Jan 26, 2026
26 checks passed

d-v-b deleted the feat/get_json branch January 26, 2026 08:52

Uh oh!

Conversation

d-v-b commented Jan 4, 2026

Uh oh!

codecov bot commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

paraseba commented Jan 4, 2026

Uh oh!

d-v-b commented Jan 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomAugspurger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-v-b Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

d-v-b commented Jan 5, 2026

Uh oh!

d-v-b commented Jan 8, 2026

Uh oh!

d-v-b commented Jan 16, 2026

Uh oh!

codspeed-hq bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Summary

Footnotes

Uh oh!

maxrjones commented Jan 16, 2026

Uh oh!

d-v-b commented Jan 16, 2026

Uh oh!

d-v-b commented Jan 20, 2026

Uh oh!

d-v-b commented Jan 23, 2026

Uh oh!

maxrjones left a comment

Choose a reason for hiding this comment

Uh oh!

maxrjones left a comment

Choose a reason for hiding this comment

Uh oh!

d-v-b commented Jan 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Jan 4, 2026 •

edited

Loading

d-v-b Jan 5, 2026 •

edited

Loading

codspeed-hq bot commented Jan 16, 2026 •

edited

Loading